“Analyzing textual data from open-ended survey questions”

Gerard Chung,
Social Service Research Centre

Survey on parents during Circuit-breaker

Impact of savings programs on youths’ money management

Benefits of open-ended survey questions

  • Capture respondents’ experiences in their own words
  • Gather new information about a topic
  • Supplement quantitative findings

Closed-ended questions is more prevalent

  • These questions offer a limited set of response options.
  • easier to ask, code, and analyze”1

It is difficult with large “messy” text data


 [1] "Asking his daughter to sign up for one of the options tells the daughter his dad does not truly understand her hobbies and interest.  Instead he should say \" Summer lets get you into tennis which I know you like playing with your friends Marcy and Kathy, you seem to enjoy the work out and we can find a league in town that can challenge your skills.\" or he can say \" Well summer you play the guitar well, mom and I would be excited to invest into lessons that work with your homework schedule, giving you time with friends but fills that gap of feeling bored.\""
 [2] "Play outside. Give them room to be kids if they have no reason to be watched like hawks, then dont."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 [3] "He showed his son they have something in common."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
 [4] "She showed empathy"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
 [5] "She is encouraging her child to be social and active. I am assuming she lives in an area where she know her neighbors and would know where Nicole is going."                                                                                                                                                                                                                                                                                                                                                                                                                          
 [6] "they about ecited that subject"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
 [7] "She encouraged her daughter to participate in extra curricular activites"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
 [8] "Giving the opportunity to discuss what the childs concerns are made me choose this rating. Communication is a two way road."                                                                                                                                                                                                                                                                                                                                                                                                                                                          
 [9] "She didn't get mad, just explained the facts although she should reprimand talking rude."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
[10] "Ok, because you have not given us enough information. For instance, does the son participate in class, what is his current grade, class size such as student to teacher ratio, what level of class is the son in, and the first response should have been, \"why do you think that you are bored?\""                                                                                                                                                                                                                                                                                  
[11] "He is taking an active approach in trying to enrich his daughter's education."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
[12] "sounds great"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
[13] "He is acknowledging her feelings and taking into account that although she has new activities with school she is probably restless from the creative activities she likely engaged in during vacation so he is providing some options she can think about and inviting her opinions"                                                                                                                                                                                                                                                                                                  
[14] "Because a lot of times kids minds will wonder and not pay attention in class. The teacher has her education the is there to obtain one."                                                                                                                                                                                                                                                                                                                                                                                                                                              
[15] "His son is probably bored from being too advanced for what he's being taught.  Parent should supplement materials to the class or discuss with teacher ideas to keep son engaged."                                                                                                                                                                                                                                                                                                                                                                                                    
[16] "It is very good that he acknowledged his son's creativity but sometimes it hinders a child if they think that they are too good at one thing.  It does not broaden their knowledge of other activities if they only focus on one topic."                                                                                                                                                                                                                                                                                                                                              
[17] "Fine"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
[18] "Kim is trying to push her child to be involved in school and if he already knows what is being taught, she is trying to push him to learn new things.  She maybe could have asked her son why is he is bored, to see if he is just not interested in math, or if he easily learns it and it truly is not challenging for him."                                                                                                                                                                                                                                                        
[19] "She offered none of her time to try to assist at home with sparking an interest."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
[20] "Well he looked beyond the stated problem for potential reasons why his son would feel the way he did. He then proffered a solution. Michael's son can still backpedal at this point if that isn't a result he wanted."                                                                                                                                                                                                                                                                                                                                                                


Topic modeling

  • A computational method to extract knowledge from large amounts of textual data
    • inductively identifies latent topics among a large collection of documents and …
    • assigns individual documents to one or more topical categories.

“What has help you cope with parenting?”

“.. what would you do to set aside enough money to buy it later?”

Challenges

  • Potentially useful text data but hard to access
    • case notes, risk assessments, social reports etc.
  • Short text challenge
    • severe data sparsity
    • biterm topic model
  • Working with qualitative approaches
    • computational grounded theory framework (Nelson, 2020)
    • 3-step: pattern detection -> refinement -> confirmation
  • Corpus linguistics methods


Understanding the experiences of social work clients remains a hallmark goal of social work research


Find me at:

@GerardChung

@gerardchung

gerardchung.com

References

  • Geer, J. G. (1991). Do open-ended questions measure “salient” issues? Public Opinion Quarterly, 55(3), 360–370. https://doi.org/10.1086/269268
  • Thyer, B. A. (2012). The scientific value of qualitative research for social work. Qualitative Social Work, 11(2), 115–125. https://doi.org/10.1177/1473325011433928


Understanding the experiences of social work clients remains a hallmark goal of social work research


Find me at:

@GerardChung

@gerardchung

gerardchung.com